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OVERVIEW 



Background 

The Massachusetts Basic Skills Assessment of listening and 
speaking was conducted during the 1979-80 school year in response 
to the Massachusetts Basic Skills Improvement Policy , The purpose 
of the pel icy Ts to improve student competencies in the basic 
skills of mathematics, reading, writing, listening and speaking. 

As a part of the Basic Skills Improvement Pol icy , the State 
Department of Education is required to assess the current level of 
performance of public school 12th graders in the basic skills and 
to provide additional assessment instruments to schools for their 
own use. Secondary schools are then required to set their own 
standards for performance in the basic skills and to assess their 
students in these skills, using the state developed tests or 
commercially or locally developed tests which have been approved 
by the state. 

The design and implementation of the listening and speaking 
assessment was undertaken by the Education Commission of the 
States (ECS), a private, nonprofit organization located in Denver, 
Colorado, under contract with the Massachusetts Department of 
Education. The contractor was responsible for developing listening 
and speaking instruments for the statewide assessment. In 
addition, the contractor was responsible for developing three 
listening instruments and three speaking instruments, which were 
equivalent in content and difficulty to the instruments used in 
the statewide assessment. These alternate 'forms were for use by 
local school districts for their own assessments. 

This paper describes the nature of the listening and speaking 
instruments used for the statewide assessment. The alternate forms 
of the instruments are held secure by the Massachusetts Department 
of Education. The paper also briefly summarizes the results of the 
statewide survey. The instruments and results reflect the specific 
conditions present in the state of Massachusetts. Nevertheless, 
many other states and school d istr icts have current or pro j ected 
programs for listening and speaking instruction and assessment and 
may find the Massachusetts project a useful model. 
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Assessment Design 



I 



The scope of the assessment was established by the listening 
and speaking objectives identified in the Basic Skills Improvement 
Policy and Regulations « These obj ectives focus on ski lis essential 
for everyday functioning in school r employment and community 
settings. They are minimal competencies that all students should 
master before leaving high school. Thus, the goal of the 
assessment is to measure the most impo rtant and basic 1 istening 
and speaking skills, not the full range of skills within these 
areas. i 

For the listening assessment, groups of students listened to 
material on a tape recording and then answered multiple-choice 
questions about what they heard. The instrument is unique in that 
the passages students 1 istened to represent typical spoken 
language such as news stories, public service announcements, 
commercials and conversations. Most existing listening tests use 
typical written material which is read aloud. 

The speaking assessment used two measurement approaches* For 
the first approach, two teachers who had a student in class 
independently assessed the student using a set of speaking rating 
scales* The teachers based their ratings of a student on their 
observations of the student's typical speaking performance during 
normal classroom activity over a period of time. For the second 
approach, a trained rater assessed the student's speaking 
performance in a one-on-one situation. The rater gave the student 
several specific speaking tasks and then rated the student's 
performance on each task using the same speaking rating scales. 

In addition to the asses smei; L of listening and speaking 
performance, background questionnaires were administered to 
students, teachers and * principals. They included questions about 
general school and student characteristics and questions about 
specific factors that might influence listening and speaking 
performance. This information was collected to provide baseline 
information prior to the implementation of the Basic Skills 
Improvement Pol icy and to help explain the performance of students 
on the listening and speaking assessment. 



Saropl ing 



The assessment used a strati fi ed , mul ti -stage sampl ing 
procedure to select students for participation in the assessment. 
This sample was designed to be representative of public high 
school seniors from four kind-of-communi ty (KOC) categories: Big 
Cities , Industrial Suburbs, Residential Suburbs and Other 
Communities. In addition, procedures were followed to ensure 
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adequate representation of minorities, areas of the state and 
sizes of schools. 

The sampling plan was a self-weighting design which called 
for the random selection of 12 schools from within each KOC. 
Within each school 60 twelfth graders were randomly selected to 
participate in the listening assessment and 18 to participate in 
the speaking assessment. Students were exempted from testing who 
were non-English speaking or handicapped (physically, mentally or 
emotionally) in such a way that they could not respond to the 
assessment. The plan assumed that the sample size would be reduced 
by approximately one-third due to ineligibility and absenteeism. 
In the actual assessment one additional school was added to make a 
total of 49 participating schools.-^ 



Test Administration 

7'he listening and speaking assessment was conducted during 
March and April 1980. A total of 2,207 students were tested in the 
listening assessment and 691 students in the speaking assessment. 
No student was assessed in both are-is. 

The testing was conducted by a test administrator hired by 
ECS. The test administrator was assisted by a school coordinator 
identified by each participating school. The test administrator 
was primarily responsible for conducting the' one-on-one speaking 
assessment. The school coordinator was primarily responsible for 
conducting the listening assessment. Testing for a school was 
completed within a single day. 



Data Analysis 

The data analysis reflected the requirements of a 
self-weighting sampling design. The estimate of performance of 
students within each KOC was the simple average of scores of the 
sampled students within each KOC. The estimate of performance of 
students statewide was computed by weighting the responses of 
students within each KOC according to their actual proportion in 
the state population* The estimate of sample variation, standard 
errors, was computed using the Taylor-series estimation procedure. 



One school was added to complement a school that was selected but 
did not have the number of seniors required by the sampling plan. 
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The results were analyzed in several ways. Performance was 
reported for individual items and ratings. The results were 
summarized by types of items and ratings and by types of passages 
and tasks so that judgments could be made concerning the relative 
performance for various listening and speaking skills and 
situations. Performance was also summarized across all items and 
ratings in the instruments. All of the above results were 
provided for students statewide and for various reporting groups. 
In addition to the reports on student performance, the results of 
the background questionnaires were tabulated. 

All of the results were aggregated across students and 
schools. No individual student or school was identified in any 
way. Thusr the results reflect the level of performance of 
students across the state. They do not indicate the level of 
performance of individual students or schools. 



THE LISTENING ASSESSMENT 



' Objectives and Assessment Design 

The listening assessment was designed to measure the 11 
listening objectives identified by the Massachusetts Basic Skills 
Improvement Pol icy . Exhibit 1 lists the 1 istening objectives and 
indicates how the listening passages and questions developed for 
the assessment are related to the objectives. Some of the 
objectives deal with general listening skills that apply to all 
listening situations — for example^ Objective A-1, Recognize 
Words and Phrases. Other objectives deal with specific listening 
situations for example. Objective C-l, Understand and Respond 

to Survival Words Used in Emergency Situations. 

To assess the listening objectives, a test was developed that 
includes a variety of listening passages and asks a ariety of 
listening questions. The text of the passages and items are 
provided in Appendix A. The passages reflect the objectives that 
deal with specific listening situations. The questions reflect the 
objectives that deal with general listening skills. The test 
measures the obj ect ives global ly; it does not measure each 
individual objective. Thus, the instrument is a general 
descriptive test of listening ability, not a diagnostic test. 



EXHIBIT L Relationship of Passages and 
Questions to Listening Objectives 



Objectives 

Basic Listening Skills 

1. Recognize words and phrases used by the speaker 

2. Indicate why the speaker can or cannot be understood 
Understanding What You Hear 

1. Understand spoken words and Ideas 

2. Identify and understand main Ideas 

3. Associate Important details with main ideas 

4. Understand descriptions of events and experiences 

5. Understand speaker's purpose 
Using What You jear 

1. Understand and respond to surl/lval words used in 
emergency situations 

2. Summarize Information and draw conclusions 

3. Recognize when words and phrases are used to 
convince or persuade 

4. Follow straightforward directions 
1 



Passages 



3 
8 

2,6,13,15,17,18 

1,U,19 

4,7,10,20,21 

News Story 

Public Service Announcement 

5,11,12,16 

Conve.-sation 

9,22 

Connnerclal 

Telephone Conversation 
Teacher's Announcement 

8 
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The Listening Test 

In the first stage of instrument development^ four general 
categories of listening situations were identified. These are 
derived from the common listening situations specified in 
Objectives B-4, C-l^ C-3 and The situations include: 

• Descriptions of events ana experiences 

• Emergency messages 

• Persuasive messages 

• Sequences of d^.rections 

Stimulus materials were obtained from radio and TV stations 
or developed by the contractor to reflect the specific situations* 
Six passages were selected for the final version of the listening 
test# Descriptions were represented by a news story and a public 
service announcement. Emergency messages were represented by a 
conversation which takes place during an emergency. Persuasive 
messages were represented by a commercial. Sequences were 
represented by a telephone conversation and a teacher's 
announcement. The number of passages chosen for each category 
reflects the relative importance of the situation for mastery of 
basic skills as judged by review committe^^ established by the 
Department of Education. 

All the listening passages represent natural spoken language 
and reflect listening experiences that are common to all students. 
They also span a variety of in-school and out-of-school listening 
situations, some of which are formal — like the news story — and 
some of which are informal — like the conversation. The 
listening stimuli are relatively short, ranging from 30 seconds to 
1-3/4 minutes. The vocabulary and grammar in the passages are 
relatively simple. These characteristics reflect the basic skills 
orientation of the assessment and help reduce the influence of 
memory and motivation on performance. 
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Multiple-choice questions were then written for the passages. 
The questions reflected the general listening skills identified in 
objectives A-1 , B-1, B-5 and C-2. The skills include: 



• Recognize words and phrases 
9 Identify problems 

• Understand words and ideas 

• Identify main ideas 

• Associate details 

• Und e r s t a nd pu r po s e 

• Draw conclusions 

Tweaty-two questions (three to four per passage) were 
selected for the final version of the listening test^ The 
questions related to each skill are indicated in Exhibit 1. The 
number of questions for each skill reflects the relative 
importance of a skill for mastery of basic skills as judged by the 
review committees. 

All of the items are stimulus dependent. They can only be 
answered by listening to the material; they cannot be answered by 
a student's previous kn'owledge or experience* 

All passages and items were field tested prior to final 
selection and were judged by the review committees to be 
culturally fair and balanced with respect to sex, racial/ethnic, 
geographic and socioeconomic representation. 

The listening test was designed to be virtually 
self-contained on a cassette tape. Students were provided with 
booklets that included instructions and the multiple-choice 
questions and with separate answer sheets. Students heard the 
instructions, then a passage and then questions about the passage. 
Questions and response options were read aloud to minimize the 
effect of a respondent' s reading ability on listening performance. 
After each item was read, there was a pause; this allowed students 
plenty of time to answer the question. They did so by marking 
their responses on an answer sheet. 



Results of the Listening Assessment 

Students" performance on the listening test was very high, 
which is to be expected in a basic skills test. On the average, 
students correctly answered 88% of the items which is equivalent 
to about 19 out of 22 items. 

Students' performance was high for all seven skills areas. 
Questions related to two skills — Recognize Words and Phrases and 
Understand Words and Ideas — appeared to be relatively harder for 
students than other questionso The average percentage correct for 
these two areas was 84%. Questions related to three skills — 
Identify Problems, Identify Main Idea and Understand Purpo/se — 
appeared to be very easy for students, with average percentages 
correct above 90% in all three areas. Students' performance on 
questions related to the remaining two skills — Associate Details 
and Draw Conclusions — fell in between the two extremes. The 
average percentage correct for these two areas was 88%. It should 
be noted, however, that there were very few items for each 
listening skill. In two cases there was only one item for a skill. 
This makes it inappropriate to place much emphasis on the relative 
difficulty of the different listening skills. 

Performance was high for all listening situations except for 
Persuasive Messages which was much lower. The average percentages 
correct for Descriptions, Emergency Messages and Sequences were 
88% or higher. The average percentage correct for Persuasive 
Messages was 73%, a dramatic difference. In a previous 
Massachusetts Basic Skills Assessment of reading, similar results 
were obtained. Students' performance was noticeably lower on 
questions about persuasive reading passages than on questions 
about other types of reading material. This result suggests that 
persuasive situations are particularly difficult for students. 
However, it should be noted that this situation was only 
represented by one passage and three questions. Also, one of the 
questions (question 7) was unusually difficult and required 
students to draw a very high-level inference about the material. 
Thus, the results are suggestive of a problem area but not 
definitive. 

Reliability and Validity of the Listening Test 

In addition to the analysis of student performance, several 
steps were implemented prior to and during the assessment to test 
the reliability and validity of the listening test. 

The reliability of the test was determined by a test of the 
internal consistency of the items. The Hoyt estimate of 
reliability yielded a correlation coefficient of .76. 
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Prior to the assessment, the content validity of the 
listening test was determined by expert judgment, A panel of 
communication experts independently examined the listening 
passages and items and in a blind review matched them to the 
listening objectives. In general the panel agreed almost 
unanimously with respect to the categorization of passages and 
agreed most of the time with respect to the categorization of 
items. Based on the reviewers' comments, adjustments were made in 
the test so that it more nearly reflected the objectives. 

The review committees established by the Department of 
Education felt that assessment procedures results confirmed the 
reliability and validity of the listening test. 



THE SPEAKING ASSESSMENT 



Objectives and Assessment Design 

The speaking assessment was designed to measure the 14 
speaking objectives identified by the Massachusetts Basic S kills 
Improvem ent Policy . Exhibit 2 lists the speaking objectives and 
indicates how the speaking rating scales and tasks developed for 
the assessment are related to the objectives. Similar to the 
listening objectives, some of the speaking objectives deal with 
general speaking skills that apply to all speaking situations — 
for example. Objective A-1, Use Words and Phrases Appropriate to 
the Situation. Other objectives deal with specific speaking 
situations — for example. Objective C-1, Use Survival Words to 
Cope with Emergency Situations. 

A set of speaking rating scales were developed for the 
speaking assessment. These scales were used in two ways: 

• For classroom observation by teachers 

• For one-on-one assessment by trained raters 

The following sections describe the speaking rating scales, 
the teacher observation approach, the one-on-one approach and how 
the approaches are combined into a two-staged assessment. 
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EXHIBIT 2. Relationship of Tasks and Ratings 
to Speaking Objectives 



Objectives 

A. Basic Oral Communication Skills 

1. Use words and phrases appropriate to the situation 

2. Speak loudly enough to be heard by a lister.er or group 
of listeners 

3. Speak at a rate the listener can understand 

4. Say words distinctly 

B. Planning, Developing^ and Stating Spoken Messages 

1. Use words in an order that clearly expresses the 
thought 



Tasks 



Ratings 



H 

o 



2. Organize main ideas for presentation 

3. State main ideas clearly 

4. Support main ideas with important details 

5/ Demonstrate knowledge of standard English usage 
C. Common Uses of Spoken Messages 

1. Use survival words to cope with emergency 
situations 

2. Speak so listener understands purpose 

3. Ask for and give straightforward information 

4. Describe objects, events and experiences 

5. Question others' viewpoints 



Respond to an Emergency 



Explain a Sequence 

Describe an Object, Event 
or Experience 

Persuade Someone to Do 
Something 



Content 

Delivery 
Delivery 
Delivery 



Organization 

Organization 

Content 

Content 

Language 



Content 
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speaking Rating Scales 

A set of four scales was developed to measure the objectives 
that deal with general speaking skills. Each scale measures one 
dimension of speaking skills. The dimensions include: 

• Delivery 

• Organization 

• Content 

• language 

The delivery dimension focuses on Objectives A-2, A-3 and A-4 
and is concerned with how well a student transmits messages. It 
measures how well a student uses appropriate volume, rate and 
articulation while speaking. The organization dimension focuses on 
Objeectives B-1 and B-2 and is concerned with how well a student 
structures messaages. It measures how well a student expresses 
the sequence or relationships of ideas. The content dimension 
focuses on Objectives A-1, B-3, B-4 and C-2 and is concerned with 
how well a student provides an adequate amount of relevant 
information to meet the requirements of various speaking tasks. In 
addition, it measures how well a student adapts the content of 
messages to specific listeners and situations. The language 
rating focuses on Objective B-5 and is concerned with how well a 
student uses appropriate grammar and vocabulary while speaking. 

Within each speaking dimension, performance is rated using a 
4-point scale: 

1 = Inadequate 

2 = Minimal 

3 = Adequate 

4 = Superior 

The dimensions and the levels of the rating scales are 
explained in the the "Speaking Assessment Ratings Guide" which may 
be found in Appendix B. 

The Teacher Observation Approach 

The teacher observation approach is a general measure of a 
student's speaking performance. In this' approach, two teachers who 
had the same student currently enrolled in class independently 
rated the student's general speaking performance in class using 
the speaking rating scales. Usually one of the teachers was an 
English teacher and the other from another subject area where 
students participate in a fair amount of classroom discussion and 
interaction, such as history, government or science. 
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Teachers were asked to read the "Speaking Assessment Ratings 
Guide" and then to complete the ratings. Teachers based their 
ratings of a student on their observation of the student's 
performance in normal classroom activities, ouch as asking 
questions, responding to que'-tions, explaining how to do 
something, giving a report to the class or talking with other 
students in discussion groups. They considered the student's 
average performance since the beginning of the semester. 



The One-on-One Approach 

The one-on-one approach Is a focused measure of a student's 
speaking performance. In this approach, a trained rater rated the 
student* s performance using the same speaking rating scales used 
by teachers for the teacher observation approach. However, 
Instead of basing ratings on classroom observations, the rater 
gave the student specific tasks and rated the student's 
performance on these tasks. 

Each rater was prov ided wl th one day of training in the 
one-on-one approach. The rater assessed each student 
individually. The rater gave the student several speaking tasks 
and rated the student's performance on each task along the four 
dimensions. Thus, for each task, the rater gave the student a 
rating from 1 to 4 for delivery, organization, content and 
language. The rating was assigned immediately after the student's 
response. 

The speaking tasks used in the one-on-one approach reflect 
the objectives that deal with specific speaking situations. The 
tasks include: 

• A description 
e An emergency 

• A sequence 

o A persuasion task 

The description task focuses on Objective C-4 and is 
concerned with how well a student can describe an object, event or 
experience so that another person would know something about the 
topic . The emergency task focuses on Ob j ective C-1 and is 
concerned with how well a student can prov ide the necessary 
information in '-*n emergency so that another person could send 
help. The sequence task focuses on Objective C-3 and is concerned 
with how well a student can explain a sequence of steps so that 
another person could follow the sequence. The persuasion task 
focuses on Objective C-5 and is concerned with hov well a student 
can present effective arguments so that another person v/ould be 
persuaded by the student's point of view. 
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The focus of all of the speaking tasks was on the student^ s 
effectiveness in transmitting the message — not on the specific 
content of the messages. Tasks were developed which were familiar 
to all students and did not require any special knowledge or 
experience. The tasks were field tested with 9th graders and 12th 
graders from inner city, suburban and rural schools to assure that 
they would be relevant for a wide variety of students of various 
ages. Based on field test results, four tasks, one of each type, 
were selected for the final version of the speaking test. These 
tasks were judged to be substantially free of sex, racial/ethn ic 
geographic and socioeconomic bias by the review committees. The 
text of the tasks is provided in Appendix C. 



The Two-Staged Approach 

The rationale for developing a two-staged approach for 
assessing speaking is based on the need for measures that are 
reliable, valid and free of bias. The teacher observation approach 
provides a general measure of student performance. It assesseis 
all types of speaking tasks as they occur in a natural situation^ 
However, sometimes a general measure such as this one allows for 
other factors, such as academic achievement or sociability, to 
enter into the ratings. The one-on-one approach provides a focused 
measure of student performance. It assesses only a few types of 
speaking tasks in a contrived setting* However, it focuses 
entirely on speaking variables and uses standardized procedures. 
The two approaches complement one another and taken together guard 
against the many problems of reliability, validity and bias that 
are inherent in speaking measures. 

Another reason for the two-staged approach is the need for 
measures that are feasible for large-£<cale use by school districts 
in the future. The intention is to use the teacher observation 
approach as a screening measure and to use the one-on-one approach 
as a back-up measure in cases where a student's level of ability 
is in question. School districts would use the teacher observation 
measure to assess all students in a grade. They would use the 
one-on-one measure to assess students whose level of ability is in 
question. For example, the one-on-one measure might be used when 
two teachers do not agree in their observation ratings of a 
student . 

Before final plans are made for implementing the speaking 
measures in school districts, it is necessary to demonstrate that 
the two approaches are reliable and valid. Initial tests of the 
reliability and validity of the measures were conducted during the 
statewide assessment. The review committees felt that initial 
results were encouraging. However, further study will be 
undertaken before final recommendations are made regarding how the 
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measures should be ustd by school districts. 

Results of the Speaking Assessment 

On the average, students performed adequately in all areas of 
the speaking assessment. Thei r average ratings for the teacher 
observation approach and the one-on-one approach were about the 
same, both falling just below 3 on a scale of 1 to 4. 

The average rating of students for the teacher obse rvation 
ratings overall was 2.95. The averages for each speaking dimension 
ranged from 2.91 for Delivery to 2.99 for Content. Thus, the 
level of performance did not vary much from dimension to 
dimension • 

The average rating of students on the one-on-one ratings 
overall was 2.89, slightly lower than the average rating for the 
teacher observation ratings. ,The ratings summarized across 
speaking dimensions ranged from 2.85 for Content to 2.94 for 
Language. The ratings summarized across speaking tasks ranged from 
2.84 for Description to 2.97 fo^ Emergency. Thus, similar to the 
teacher observation results, the level of performance on the 
one-on-one ratings did not vary much from dimension to dimension 
or from task to task. 

Most of the ratings (79%) assigned to students in the 
one-on-one assessment were 3's which reflect adequate performance. 
Very few ratings (about 1%) were I's for inadequate performance 
and few ratings (about 6%) were 4's for superior performance. Of 
interest is the small but noticeable percentage of 2 ratings 
(about 17%) which reflect minimal performance. Another result 
which demonstrates a lack of skills among some students is the 
failure of 18% of the students to mention the nature and location 
of the emergency in the emergency task. 

Reliability and Validity of the Speaking Approaches 

In addition to the analysis of student perfo rmance, several 
steps were implemented prior to and during the assessment to test 
che reliability and validity of the speaking approaches. 

The reliability of the teacher observation approach was 
determined by examining the consistency of the ratings of two 
teachers for the same students. Approximately 95% of the ratings 
were either identical or adjacent (wi thin one point of one 
another). There were no systematic differences in the ratings of 
different types of teachers (English teachers compared to other 
teachers) . 
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The reliability of the one-on-one approach was determined by 
examining the consistency of the ratings of the trained raters for 
the same student. In a test of inter-rater reliability conducted 
prior to the assessment, raters assigned identical ratings 85% of 
the time. Rater consistency was determined after the assessment 
by rescoring 10% of the student responses* Rescoring was conducted 
by project staff who were reliable raters. In this test, raters 
assigned identical ratings 75% of the time. 

The validity of the speaking approaches was determined by 
expert judgment and empirical tests. Prior to the assessment, a 
panel of communication experts independently examined the speaking 
tasks and speaking dimension and in a blind review matched them to 
the speaking objectives. In general the panel agreed almost 
unanimously with respect to the categorization of tasks and most 
of the time with respect to the categorization of speaking 
dimensions. Based on the reviewers' comments, adjustments were 
made in the tasks and dimensions so that they more nearly 
reflected the objectives. 

The degree to which the two assessment approaches were 
measuring the same thing, concurrent validity, was determined by 
comparing the teacher observation ratings and one-on-one ratings 
for the same student. Ratinjs were combined to form a total 
teacher observation score and an adjusted total one-on-one score, 
both scores ranging from 8 to 32. In 81% of the cases the scores 
were within 4 points of one another and in 98% of the cases scores 
were within 8 points of one another. 

The review committees established by the Department of 
Education felt that the assessment procedures and results 
confirmed the reliability and validity of the two-staged speaking 
assessment approach. 



SUMMARY 



The Massachusetts Basic Skills Assessment of listening and 
speaking provides viable models for measuring listening and 
speaking skills of secondary level students and reveals 
interesting results regarding levels of competence of students in 
these areas. The instruments and results reflect the specific 
conditions present in the state of Massachusetts. Nevertheless, 
the Massachusetts experience should be useful to the increasing 
number of states and school districts which are moving into 
instruction and assessment of listening and speaking skills. 
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APPENDIX A 



NEWS STORY 

You are about to hear a recording of a news story. After the story you will be asked several 
questions about what you heard. 

Do not take notes while you are listening to the recording. 
Do not turn the page until told to do so. Now listen carefully. 



The artistic process is a complex one - take song w riting for example. You picture the songwriter, 
sitting alone at the piano, composing some melodic "work of art. But a Philadelphia firm has come 
up with a way of making that songwriter's job somewhat easier. Charles Trois of Tenel industries 
has devised a computer system which he claims can help a songwriter pick a hit melody. To 
compute a song's hit potential, the melody is fed into the computer and within ten minutes the 
song is rated on a scale of one to ten. Trois explains his rating system as such: A song with a score 
of one to three will be a flop; a four means it's a possible hit; songs rated five and six could make 
the charts, and a seven might make the top ten. Melodies carrying ratings of eight and nine will be 
number one hits while a ten should becpme a classic. Trois claims that his computer has an 
eighty-two percent success rate in picking hits. However, there's only one problem with the machine 
- it can only handle pop and soft rock songs. Hard rock just won't compute..*. 




1. What is the main idea of this story? 

A. Some songs are very hard to write. 

B. Computers are now being used to pick hit songs, 

C. Computers can be programmed to write music. 

D. Most songs written will not be classics. 

2. What can Trois' machine handle? 

A. Pop and soft rock 

B. Hard rock 

C. Classical music 

D. All time favorites 

3 This story says that "a Philadelphia firm has come up with a way of making the songwriter's 
job somewhat easier." What does firm mean in the story? 

A. Computer 

B. Inventor 

C. Lawyer 

D. Company 

4. How is a song rated? 

A. By asking well known songwriters 

B. By singing the song for an audienoe 

C. By making the computer play the song 

D. By feeding the melody into the computer 
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COMMERCIAL 

You are about to hear a recording of a commercial. After the commercial you will be asked several 
questions about what you heard. 

Now listen carefully. 



Enjoy delicious fried chicken right at home this week-end.. .and save money.. .with a fresh whole 
fryer from Food King. Roast a whole fryer on the barbecue.. .split a fryer and broil the halves.. .cut 
up a fryer for fried chicken. Save on USDA Grade "A" whole fresh fryers, 10«i off our regular low 
price per pound. 
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What is the main purpose of this announcement? 

A. To convince people that Food King sells only top quality chicken 

B. To give people information about food prices 

C. To tell people how they can cook chicken 

D. To persuade people to buy chicken at Food King 

What did the announcement say about the price of chicken at Food King? 

A. Chicken is 1 0<i off per pound. 

B. Chicken is 10% off. 

C. Chicken is 2Qi off per pound. 

D. Chicken is at its every day low price. 

Why should people buy chicken now at Food King? 

A. Because chicken is delicious when fried 

B. Because Food King has fresh fryers 

C. Because chicken is on sale 

D. Because Food King fryers are cut up 
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TELEPHONE CONVERSATION 

You are about to hear a recording of a telephone conversation. After the conversation you will be 
asked several questions about what you heard. 

Now listen carefully. 



Child: Hello? 

Adult: Hello. I'd like to speak to Mrs. Kaplan. 

Child: She's not here. 

Adult: Oh, (pause) do you expect her back soon? 

Child: No, she's out of town. She'll be back on Friday. 

Adult: Ok, would you give her a message? I 

Child: Yeah, wait a minute. I've got to get something to write with. (Paused Ok. 

Adult: This is Mrs. Benton. I'm cafling to remind your mother of the Neighborhood Action 
Group meeting this Saturday night. Tell her we'll need the box of pencils because we'll be 
voting. Also, she promised to give me a ride so I'll be expecting her to pick me up. Oh, 
yes, one more thing, we've got to be at the meeting at least 10 minutes early. Did you get 
all of that? 

Child: Yeah, I gol; it. Do you want Mom to call you? 

Adult: Yeah, I think that would be a good idea. I might think of something else before then. 
(This last sentence is spoken softly and mumbled.) 

Child: What? 

Adult: I said I might think of something else before then. 

Child: Oh, Ok. 

Adult: Thanks so much. 

Child: Ok, bye. 

Adult: Goodbye. 



8. Why did Mrs. Benton have to repeat part of her message? 

A. Because there was a loud noise in the background 

B. Because she started to cough 

C. Because she spoke too softly and mumbled 

D. Because she talked too fast 

9. What will Mrs. Kaplan probably do as soon as she returns home? 

A. Go to the meeting 

B. Go to her office for messages 

C. Type up the ballot for voting 

D. Call Mrs. Benton 

10. Why should Mrs. Kaplan bring pencils to the meeting? 

A. To get addresses of new people 

B. To have for taking attendance 

C. To write down the message 

D. To have for voting 

1 1 . Why did Mrs. Benton call Mrs. Kaplan? 

A. To remind Mrs. Kaplan of the Neighborhood Action Group meeting 

B. To find out about Mrs. Kaplan's trip 

C. To give Mrs. Kaplan directions to her house 

D. To encourage Mrs. Kaplan to vote at the Neighborhood Action Group election 
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TEACHER'S ANNOUNCEMENT 



You are about to hear a recording of a teacher's announcement. After the announcement you will 
be asked several questions about what you heard. 



Now listen carefully. 



Now, if I can have your attention for a few minutes, I'd like to explain a new assignment. Please 
settle down so everyone can hear. 

After we finish our currer \ ^it, we'll have one free class period before break. What I'd like to do 
with this time is to hav€ - party, but this won't be an ordinary party. It will be a special party 
because we're going to invite people from the past. And each one of you will be one of those 
historical people. During the party, you'll be asked many questions about your life and what you 
did that was important, so this will require a little bit of preparation. Tomorrow we'll spend some 
time in class drawing up a list of names of famous, historical people. This way we'll get a lot of 
different ideas. Next, we'll put all of the names in a box and each draw one name. Then, before the 
party, you'll have to learn about the person whose name you drew. (Pause) So, if there are no 
questions, I'll go on with today's topic. 
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1 2. What is the main purpose of the teacher's announcement? 

A. To encourage students to read magazines and newspapers 

B. To explain a new assignment to the class 

C. To remind students about school break 

D. To urge students to always come to class well prepared 

13. What will the class study before break? 
A. People from the past 

3. Political parties 

C. Social customs 

D. Current events 

14. What are the students supposed to do? 

A. Find out about a famous person 

B. Go to a costume party 

C. Write about a fanrious person 

D. Collect pictures of famous people 

1 5. When will the students draw up a list of names? 

A. Before the end of the day 

B. Tomorrow 

C. Next week 

D. Before break 
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PUBLIC SERVICE ANNOUNCEMENT 



You are about to hear a recording of a Public Service Announcement. After the announcement you 
will be asked several questions about what you heard. 

Now listen carefully. 



Your City Parks and Recreation Department has something for you every season. Register now for 
the final fall session of Outdoor Nature Programs. A variety of wildlife, ecology, and conservation 
classes and activities are offered through November for the whole family. Young Explorers meet in 
the morning or after school and teens may become wildlife volunteers. Nature hikes and cross 
country ski trips are available for all ages and ability levels. For more information, call us at 
472-4523. 
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16. What is the main purpose of this announcemer t? 

A. To urge people to donate money to the Outdoor Nature Program 

B. To inform people about current problems of ecology 

C. To persuade people to protect birds and other wildlife 

D. To give people information about nature classes 

1 7. Who are the classes for? 

A. Residents only 

B. Club members 

C. Young Explorers only 

D. The whole family 

18. Who is sponsoring the nature programs? 

A. The Young Explorers 

B. A local conservation group 

C. Wildlife volunteers 

D. The Parks and Recreation Department 
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CONVERSATION 



You are about to hear a recording of a conversation. After the conversation you will be asked 
several questions about what you hejard. 

Now listen carefully. 



Janipe: 
Man on Bus: 

Janice: 

Passenger: 

Janice: 

Passenger: 

, Janice: 

Passenger: 

Janice: 
Passenger: 
Man on Bus: 

Janice: 
Man: 
Janice: 
Passenger: 



I always enjoy this bus ride up to Vermont. It's so good to get away from the 
city. 

Yes, everything seems so much cleaner out here. ! don't know, though, sometimes 
these buses make me feel a little queasy, all the shaking back and forth. They're 
always so stuffy. I'm beginning to feel a little dizzy right now. I think...(pause). 

Hey, hey! Are you all right? Somebody, help me! He's passed out. Stop the bus! 

What's going on? What's the matter with him? 

I think he's fainted. He said something about feeling dizzy. Could you help me? 
What should we do? 

Sure. Let's lower his head between his knees. Get his hands out of the way. Yeah, 
that looks good. He'll be all right in a minute. 

Good, 'cause I don't know if I can hold him like this for long. When he said he 
felt dizzy, he really meant it! 

Yeah, he looks really pale. I think we'll be getting off the freeway at the next 
exit. Do you think you can hold him by yourself for a minute while I go ask the 
driver? 

Yes, I can hold him a little bit, but don't take too long. 
OK, I'll be right back. (Pause) 
Mmm...Hmmm... What happened? 

Don't try to sit up yet Keep you» head down. Are you feeling better? 

Yeah, but I feel a little shaky. I guess I passed out. 

Yeah, you did, but don't tr/ to talk. Just be quiet, 'til you feel better. 

Oh, I see he's coming around. Good. The driver says we'll be stopping in a minute. 
He'll feel better then. 
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19. What happened to the man on the bus? 

A. He stood up and bumped his head. 

B. He stopped breathing. 

C. He had a mild heart attack. 

D. He fainted. 

20. Why did the man on the bus feel queasy? 

A. Because the air was polluted 

B. Because he hadn't eaten' all day 

C. Because he was taking medication 

D. Because the bus was shaking back and forth 

21. What was the first thing the girl and the woman did to help the man? 

A. They opened the window on the bus. 

B. They lowered the man's head between his knees. 

C. They shook the man by his shoulders. 

D. They asked the man if he wanted a drink of water. 

22. What probably happened after the woman talked to the driver? 

A. The bus slowed down until the man felt better. 

B. The man passed out again. 

C. The woman called the police. , 

D. The bus driver stopped as soon as possible. 
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MASSACHUSETTS DEPARTMENT OF EDUCATION ASSESSMENT OF BASIC SKILLS 
SPEAKING ASSESSMENT RATINGS GUIDE 



OVERVIEW 

There are numerous kinds of speaking tasks that students must perform in everyday life both in 
school and out of school. The Massachusetts Basic Skills Improvement Policy has focused'on some 
of these tasks, including describing objects, events and experiences, explaining the steps in a 
sequence, providing information in an emergency and persuading someone. 

In order to accomplish a speaking task, the speaker must formulate and transmit a message to a 
listener. This process involves deciding what needs to be said, organizing the message, adapting the 
message to the listener and situation, choosing language to convey the message and finally delivering 
the message. The effectiveness of the speaker may be rated in terms of how well the speaker meets 
the requirements of the task. 

The Massachusetts test of basic skills in speaking separates speaking skills into four dimensions: 

Delivery 
Organization 
Content 
Language 

Delivery is concerned with the transmission of the message, i.e., volume, rate and articulation 
Organization ts concerned with how the content of the message is sequenced and how the ideas are 
related to one another. Content is concerned with the amount and relevance of information in the 
nnessage, and how the content is adapted to the listener and situation. Language is concerned with 
the grammar and words which are used to convey the message. 

Each of the four dimensions is rated on a four point scale: 1 is the lowest rating and 4 is the high est 
rating. A general set of principles underlies the rating scale for all four components. Ratings of 1 
reflect speaking skills which are inadequate in meeting the requirements of the task, natlngs of 2 
reflect speaking skills which are minimal in meeting the requirements of the task. Ratings of 3 
reflect speaking skills which are adequate in meeting the requirements of the task. Ratings of 4 are 
superior in meeting the requirements of the task. 

Iridividuals who act as raters for the speaking assessment need to take the role of a naive objective 
listener. The rater must be naive so that the rater can base his or her rating on exactly what the 
speaker says. The rater must be careful not to let his or her own knowledge and experience 
influence the rating. The rater must face each speaker as if it were a new experience. The rater must 
also be objective so that he or she does not let a particular set of norms pf social acceptability 
influence the rating. The rater must evaluate the speaker in terms of how well the speaker meets the 
requirements of the speaking task, irrespective of the particular communication style the soeaker 
uses. 
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DELIVERY 



The delivery rating focuses on the transmission of the message. It is concerned with volume , rate 
and articulation> Articulation refers to pronunciation and enunciation. Some examples ot poor 
articulation include mumbling, slurring words, stammering, stuttering and exhibiting disfluencies 
such as ahs, uhms or "you knows." 

1 = The delivery is inadequate in meeting the requirements of the task. 

e.g., The volume is so low that you canno'c understand most of the message. 
' The rate is so fast that you cannot understand most of the message. 
The pronunciation and enunciation are so unclear that you cannot understand most of 
the message. 

2 = The delivery is minfmal in meeting ti^e requirements of the task. 

e.g.. The volume is too low or too loud. 

The rate is too fast or too slow. Pauses are too long or at inappropriate spots. 

The pronunciation and enunciation are unclear. The speaker exhibits many disfluencies 

such as ahs, uhms or "you knows." 

You are distracted by problems in the delivery of the message. 

You have difficulty understalnding the words in the message. You have to work to 
understand the words. 

3 = The delivery is adequate in meeting the requirements of the task. 

e.g.. The volume is not too low or too loud. 

The rate is not too fast or too slow. Pauses are not too long or at inappropriate spots. 
The pronunciation and enunciation are clear. The speaker exhibits few disfluencies, such 
as ahs, uhms and "you knows." 

You are not distracted by probliems in the delivery of the message. 
You do not have difficulty understanding the words in the message. 

4 = jhe delivery is superior in meeting the requirements of the task. 

e.g.. The speaker uses delivery to emphasize and enhance the meaning of the message. Tht 
speaker delivers the message in a lively, enthusiastic fashion. 
The volume varies to add emphasis and interest. 
Rate varies and pauses are used to add emphasis and interest. 

Pronunciation and enunciation are very clear. The speaker exhibits very few disfluencies 
such as ahs, uhms or "you knows." ' 

NOTE: In articulation you may be concerned with accent However, articulation should be rated 
with respect to your ability to understand the message, not the social acceptability of the accent 
One particular accent is not considered better thaii another REMEMBER, in this component you 
are rating how the student speaks, not what the student says. 
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ORGANIZATION 



The organization rating focuses on how the content of the message is structured. It is concerned 
with sequence and the relationships among the ideas in the message. 



1 s The organization is inadequate in meeting the requirements of the task. 

e.g., The message is so disorganized that you cannot understand most of the message. 

2 = The organization is minimal in meeting the requirements of the task. 

e.g., The organization of the message is mixed up; it jumps back and forth. 
The organization of the mes:nage appears random or rambling. 

You have difficult/ understanding the sequence and relationships among the ideas in the 
message. Vou have to make some assumptions about the sequence and relationships of 
ideas. 

You cannot put the ideas in the message into an outline. 

3 = The organization is adequate in meeting the requirements of the task. 

e.g., The message is organized. 

You do not have difficulty understanding the sequence and relationships among the ideas 
in the message. You do not have to make assumptions about the sequence and 
relationships of ideas. 

You can put the ideas in the message into an outline. 



4 = The organization is superior in meeting the requirements of the task. 

e.g., The message is overtly organized. 
' The speaker helps you understand the sequence and relationships of ideas by using 
organizational aids such as announcing the topic, previewing the organization, using 
transitions and summarizing. 

NOTE: Make sure you are not unconsciously ''filling in" organization for a speaker, because you 
happen to know something about the speaker's topic. If you have to make assumptions about the 
organization, this fact should be reflected in your rating, REMEMBER, in this component you are 
rating how the student organizes the message, not what the student says. 




CONTENT 

The content rating focuses on the specific things which are said. It is concerned with the amount of 
content related to the task, the reievance of the content to the task and the adaptation of the 
content to the listener and the situation. 

1 = The content i$ inadequa te in meeting the requirements of the task. 

e.g.. The speaker says practically nothinc,. 

The speaker focuses primarily on irrelevant content. 

The speaker is highly egocentric. The speaker appears to ignore the listener and the 
situation. 

2 = The content is minimal in meeting the requirements of the task. 

e.g., The speaker does not provide enough content to meet the requirements of the task. 
The speaker includes some irrelevant content. The speaker wanders off the topic. 
The speaker adapts poorly to the listener and the situation. The speaker uses words and 
concepts which are inappropriate for the knowledge and experiences of the listener (e.g., 
slang, jargon, technical language). The speaker uses arguments which are self-centered 
rather than other-centered. 

3 = The content is adequate in meeting the requirements of the task. 

e.g.. The speaker provides enough content to meet the requirements of the task. 

The speaker focuses primarily on relevant content. The speaker sticks to the topic. 
The speaker adapts the content in a general way to the listener and the situation. The 
speaker uses words and concepts which are appropriate for the knowledge and experience 
of a general audience. The speaker uses arguments which are adapted to a general 
audience. 

4 ^ The content is superior in meeting the requirements of the task. 

e.g.. The speaker provides a variety of types of content appropriate for the task, such as 
generalizations, details, examples and various forms of evidence. 

The speaker adapts the content in a specific way to the listener and situation. The speaker 
takes into account the specific knowledge and experience of the listener, adds 
explanations as necessary and refers to the listener's experience. The speaker uses 
arguments which are adapted to the values and motivations of the specific listener. 

NOTE: This rating is concerned with content in terms of quantity , relevance and adaptation . It is 
not concerned \Wth content in terms of accuracy . Concerns ^ith accuracy of content fail outside a 
speaking skills assessment Also, make sure you are not unconsciously '"filling in"' content for a 
speaker because you happen to know something about the speaker's topic. If you add information, 
this fact should be reflected in your rating. REMEMBEfli, in this component you are rating the 
quantity, relevance and adaptation of what the student says, not the accuracy of what the student 
says. 
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LANGUAGE 



The language rating deals with the language which is used to convey the message. It is concerned 
with grammar and choice of words. 



1 = The language is inadequate in meeting the requirements of the task* 

e.g,. The grammar and vocabulary are so poor that you cannot understand most of the 
message. 

2 = The language is minimal in meeting the requirements of the task. 

e.g.. The speaker makes many grammatical mistakes. 

The speaker uses very simplistic, bland language. The speaker uses a "restricted code," a 
style of communication characterized by simple grammatical structure and concrete 
vocabulary. 

3 =5 The language is adequate »n meeting the requirements of the task 

e.g., The speaker makes few grammatical mistakes. 

The speaker uses language which is appropria):e for the task, e.g., descriptive language 
when describing, clear and concise language when giving information and explaining, 
persuasive language when persuading. The speaker uses an "elaborated code," a style of 
communication characterized by complex gramm.atical structure and abstract vocabulary. 

4 = The language is superior in meeting the requirements of the task. 

e.g., The speaker makes very few grammatical mistakes. 

The speaker uses language in highly effective ways to emphasize or enhance the meaning 
of the message. As appropriate to the task, the speaker uses a variety of language 
techniques such as vivid language, emotional language, humor, imagery, metaphor, simile. 

NOTE: In language you may be concerned with students who come from backgrounds where a 
foreign language or a non-standard form of English is spoken. However, language should be rated 
with respect to your ability to understand the message, not the social acceptability of the 
communication style, if a speaker's use o^ incorrect or non-standard English grammar interferes 
with your ability to understand the message, this ^3ct should be reflected in your rating, 
REMEMBER, in this component you are rating how the student conveys the message through 
language, not what the student says. 
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MASSACHUSETTS DEPARTMENT OF EDUCATION ASSESSMENT OF BASIC SKILLS 

ONE-ON-ONE SPEAKING TASKS 



Description Task ; 

Think about your favorite class or extracurricular 
activity in your school* Describe to me everything you 
can about it so that I will know a lot about it. (How 
about something like a school subject, a club or a 
sports program.) 

Emergency Task : 

Imagine that you are home alone and you smell smoke* 
You call the fire department and I answer your call* 
Talk to me as if you were talking on the telephone. 
Tell me everything I would need to know to get hjalp to 
you. (Talk directly to me; begin by saying hello.) 

Sequence Task ; 

Think about something you know how to cook. Explain to 
me step by step how to make it. (How about something 
like popcorn, a sandwich or eggs.) 

Persuasion Task ; 

Think about one change you would like to see made in 
your school, like a change in rules or procedures. 
Imagine I am the -"^'t^r incipal of your school. Try to 
convince me that the school should make this change. 
(How about something like a change in the rules about 
hall passes or the procedures for enrolling in courses.) 
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